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(54) Locating and aligning embedded images 

(57) Apparatus and methods useful to automate the 
task of locating a photograph or other image that is em- 
bedded within a larger image. An edge curve is found 
that approximates the perimeter of the embedded image 
and from the edge curve the rectangle of the embedded 
image, or a rectangle covering the embedded image, is 
calculated by processing density profiles of the edge 
curve taken along two axes. Both location and orienta- 
tion of an embedded image can be calculated. The lo- 
cation of the four comers of a rectangular embedded im- 
age can be calculated, which enables automatic crop- 
ping and rotation of the embedded image, even if fewer 
than all four comers are visible. A rectangle covering the 
image can be calculated, including a rectangle aligned 
with the axes of a larger embedding image such as is 
created when scanning a small photograph on a flatbed 
scanner. The invention operates well on rectangular im- 
ages, as well as on images having polygonal and other 
border shapes, such as circles and ellipses : including 
images with irregular shapes. 
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D script ion 

BACKGROUND OF THE INVENTION 

[0001] The present invention relates to computer-im- 
plemented image processing to find embedded images 
within raster images. 

[0002] A common application tor scanners is to scan 
and digitize photographs. Often, the scanned and digi- 
tized photograph is smallerthan the scanner image size, 
so it is embedded in a larger digitized image, from which 
the embedded photograph image will have to be extract- 
ed before it is used. Often, too, a photograph placed in 
a scanner will be tilted with respect to the scanning axes 
and so the digitized photograph will be somewhat mis- 
aligned, requiring that it be rotated as well as cropped 
from the larger image. The user is generally interested 
in removing any border around the digitized photograph 
and rotating the digitized photograph so that it is aligned 
with horizontal and vertical axes parallel to the edges of 
the photograph. 

SUMMARY OF THE INVENTION 

[0003] The apparatus and methods of the invention 
automate the task of locating a photograph or other im- 
age that is embedded within a larger image. The inven- 
tion finds an edge curve that approximates the perimeter 
of the embedded image and from the edge curve calcu- 
lates the rectangle of the embedded image or a rectan- 
gle covering the embedded image by processing densi- 
ty profiles of the edge curve taken along two axes. The 
invention can provide both location and orientation of an 
embedded image. It can provide the location of the four 
corners of a rectangular embedded image, which ena- 
bles automatic cropping and rotation of the embedded 
image, even if fewer than all four comers are visible. It 
can provide a rectangle covering the image, including a 
rectangle aligned with the axes of a larger embedding 
image such as is created when scanning a small photo- 
graph on a flatbed scanner. The invention operates well 
on rectangular images, as well as on images having po- 
lygonal and other border shapes, such as circles and 
ellipses, including images with irregular shapes. 
[0004] In general, in one aspect, the invention fea- 
tures apparatus for finding an image embedded in a 
larger image. The apparatus includes means for finding 
an edge curve of the embedded image in the larger im- 
age; means for calculating a rotation angle of the em- 
bedded image from the edge curve; and means for cal- 
culating from the edge curve a location and orientation 
of one or more line segments of a side of a rectangle 
bounding the embedded image in the larger image. Ad- 
vantageous implementations of the invention include 
one or more of the following features. Where the em- 
bedded image is an image of a rectangular photograph, 
the apparatus includes means lor locating edge seg- 
ments in the edge curve corresponding to sides of the 



embedded image in the larger image. The apparatus in- 
cludes means for locating corners of a covering rectan- 
gle of the embedded image. The apparatus includes 
means for cropping the embedded image in the frame 

5 of reference of the larger image. The means for finding 
an edge curve include means for calculating density pro- 
files on two axes of the larger image; and means for de- 
tecting edge points. The apparatus includes means for 
removing pixel noise from the larger image. 

io [0005] In general, in another aspect, the invention fea- 
tures a method of processing a digital raster image em- 
bedded in a larger digital raster image. The method in- 
cludes finding an edge curve of the embedded image in 
the larger image; calculating a rotation angle of the em- 

is bedded image from the edge curve; and calculating from 
the edge curve a location and orientation of a line seg- 
ment of a side of a rectangle bounding the embedded 
image in the larger image. Advantageous implementa- 
tions of the invention include one or more of the following 

20 features. The method includes finding the edge curve 
by calculating the positions of edge points in the larger 
image, and scanning from the perimeter into the larger 
image from starting points on all sides of the larger im- 
age and selecting as a point in the edge curve the first 

25 edge point encounter from each starting point; calculat- 
ing a rotation angle by calculating an average angle of 
the edge curve points over a windowed set of neighbor- 
ing edge curve points; rotating the edge curve by the 
complement of the calculated rotation angle; and calcu- 

30 lating the location of corners of a rectangle covering the 
embedded image. The method uses two axes that are 
perpendicular. The method includes calculating the po- 
sitions of edge points by applying a Sobel filter to the 
larger image. The method includes calculating the po- 

35 sitions of edge points by applying a Hough transform to 
the larger image. The method includes cropping the em- 
bedded image according the covering rectangle. The 
method includes removing pixel noise from the larger 
image before performing the step of finding an edge 

40 curve. 

[0006] Among the advantages of the invention are 
one or more of the following. Applied to a rectangular 
embedded image, the invention does not need all four 
corners of the rectangle to be present in the larger im- 

45 age. The invention is insensitive to noise which may be 
present in the larger image or the embedded image. The 
invention can accurately locate the outer boundary of 
the edge of the embedded image. The invention pro- 
vides a rotation angle, so the embedded image can be 

so aligned. The invention can be applied to automate the 
process of scanning in images with flatbed scanners. 
The invention can be applied to determine whether a 
photograph is present in a larger image and., if present, 
where it is. 

55 [0007] Other features and advantages of the inven- 
tion will become apparent from the following description 
and from the "claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] FIG. 1 is a flowchart of a method in accordance 
with the invention of rotating and cropping an embedded 

image. 

[0009] FIG. 2 is a flowchart expanding on a portion of 
FIG. 1. 

[0010] FIG. 3 is a flowchart expanding on a portion of 
FIG. 1. 

[0011] FIG. 4 is a flowchart expanding on a portion of 
FIG. 1. 

[0012] FIG. 5 illustrates the operation of the method 
shown in FIG. 4. 

[0013] FIG. 6 is a schematic block diagram of appa- 
ratus suitable for use of the invention. 
[0014] FIG. 7 is a flowchart of a method in accordance 
with the invention implemented in a flatbed scanner. 

DETAILED DESCRIPTION 

[0015] Referring to FIG. 1 , a method 1 00 for locating 
an embedded image operates on an input 102 that in- 
cludes a raster image in which an image is embedded. 
To simplify the terminology, the method will be described 
in terms of a larger image in which is embedded an im- 
age of a photograph. To simplify further, when the mean- 
ing is clear, the embedded image of a photograph will 
simply be referred to as the photograph. The larger im- 
age including the photograph may be the output of a 
scanner. The photograph and the larger image may be 
gray scale or color. 

[0016] In an optional initial step, a filter is applied to 
remove noise (step 1 04). A mean filter may be used to 
remove pixel noise, and is advantageous over a median 
filter for this purpose, because a median filter tends to 
remove thin lines needed by the remainder of the meth- 
od. 

[0017] In another optional initial step, brightness and 
contrast adjustments are made to the larger image (step 
106), for example, using histogram equalization. 
[0018] The method then optionally computes a den- 
sity profile along two axes, generally the vertical and 
horizontal axes of the larger image (step 108). The hor- 
izontal density profile is a sum of intensity values for all 
columns plotted as a function of row; the vertical density 
profile is a sum of intensity values, I, for all rows, x, plot- 
ted as a function of column, y: 

l(x) = £ y i(x,y) 
l(y) = Z x l(x,y) 

The density profile provides a good estimate of the lo- 
cation of the photograph. The center of mass of each 
profile provides a coordinate that lies within the photo- 
graph as long as the photograph is significantly denser 



than the surrounding image, which is generally noise 
from a scanner. The density profile may optionally be 
used to determine whether the photograph fills the entire 
image or is missing entirely from the image. The inten- 

5 sity information can optionally also be used at this point 
to determine whether, for example, there is a photo- 
graph on the scanner. In one implementation, if the 
standard deviation of the intensity of the larger image is 
less than about 10% of the range of possible intensity 

10 values, the method determines that no photograph is 
present. 

[0019] The method computes edge points in the im- 
age (step 110). Edge points are found by applying a So- 
bel filter to the image. In other implementations, other 

is edge detection methods can be used. If the results of 
applying the Sobel filter are thresholded, the resulting 
edge image is a binary image containing edges in the 
original image. In particular, the edge image contains 
the visible outer borders of the photograph. (Visible bor- 

20 ders are those that appear in the larger image.) In gen- 
eral, the edge image will also contain spurious points 
from noise as well as real points from edges within the 
photograph. 

[0020] To exclude most of the interior of the photo- 
ns graph, the method calculates the points of an edge 
curve defining a perimeter of edge points within the bor- 
ders of the larger image. In one implementation, the 
edge image is scanned both vertically (one column at a 
time) and horizontally (one row at a time) from both sides 
30 of the edge image. The edge curve is initially an empty 
list of points, which may be stored in any data structure 
that preserves the points' order. Proceeding around the 
perimeter of the image, the first edge intensity value en- 
countered in each scan that is above a threshold level 
35 is added to the list of points. This yields a list of points 
that includes spurious noise points outside the photo- 
graph, the boundary points of the photograph, as well 
as some unwanted points from within the photograph 
that appear when portions of the photograph boundary 
40 do not contain enough contrast against the larger image 
background. The number of points in the edge curve de- 
pends on the image, but is at most the perimeter of the 
larger image measured in pixels. 
[0021] Because the edge curve is obtained by travers- 
es ing the perimeter of the image in a specific order, the 
points along the edge of the photograph are ordered. 
That is, as a curve index n increases, neighboring points 
are generally encountered, with the exception of noise 
points, along the border of the photograph. 
so [0022] The rotation angle of the photograph (with re- 
spect to the axes of the larger image) is calculated (step 
120). Using the edge curve, the rotation angle of the 
photograph can be estimated with an accurate and re- 
liable calculation that is very insensitive to noise. One 
55 implementation of such a calculation, which will be re- 
ferred to as the shape-invariant angle method, or, for a 
rectangular photograph, the rectangle-invariant angle 
method. 
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[0023] FIG. 2 shows the steps of the rectangle-invar- 
iant angle method 200 for calculating the photograph ro- 
tation angle of a rectangular photograph. In this method, 
a window size is selected (step 202). This may be pre- 
determined or selected dynamically. An index n indexes 
the points of the edge curve in their order around the 
perimeter of the image (step 204). From the ordered 
edge curve, an average angle is calculated over a win- 
dowed set of neighboring indices of the selected size 
(step 206). The average angle is mapped to a shape- 
invariant angle (step 208). In the case of a rectangle- 
invariant angle, the invariant is an angle in the range of 
-45° to +45°, such that both complementary angles of a 
rectangle are mapped into the same transformed angle. 
The original angle is taken to range from -180° to 180° 
while the invariant angle lies in the range of -45° to +45°. 
This essentially calculates the original angle modulo 90° 
offset by 45°. 

[0024] As the sliding window is moved around the 
edge curve, the average angle is tabulated as a function 
of the curve point index n (step 210). This tabulated av- 
erage angle curve has regions of roughly constant angle 
and transition regions for the corners and for noise 
points. In spite of the possible presence of large 
amounts of noise, the dominant angle can be estimated 
very accurately To do this, an angle density profile is 
calculated (step 212). The angle density profile tabu- 
lates the number of occurrences of each angle for the 
angles in the range of -45° to +45°. This essentially cre- 
ates a horizontal histogram of the tabulated average an- 
gle curve. The histogram bin size may be selected ac- 
cording to the desired accuracy of the calculated angle 
value. Bin sizes of 1/2° and 1 ° have been found suitable. 
The estimated rotation angle is then calculated (step 
214). The peak density in the angle density profile is 
found and a weighted sum of its nearest neighbors pro- 
vides an accurate estimate of the rotation angle of the 
photograph. In one implementation, two nearest neigh- 
bors on each side are used. 

[0025] FIG. 3 shows the steps of a Hough-transform 
method 300 for calculating the photograph rotation an- 
gle of a rectangular photograph. This method 300 uses 
the techniques of the Hough transform to bcate the po- 
sitions and angles ot the edge segments. This is an al- 
ternative method to method 200 (FIG. 2) for implement- 
ing step 120 (FIG. 2). Returning to FIG. 3, each point of 
the edge curve is described by an angle and radius from 
an origin (step 302). The angles may optionally be a rec- 
tangle-invariant angles (step 304). A line perpendicular 
to the radius arm has the equation: 

radius = x -cos(angle) + y-sin(angle). 

The Hough transform accumulates the values for all 
possible lines in an array indexed by angle and radius 
(step 306). Every point in the original edge curve con- 
tribut s multiple points in the Hough-transformed im- 



age. For the present method, the array is index by both 
the rectangle-invariant angle and the radius (step 306). 
[0026] By accumulating the values of the in the array 
for all points in the edge curve, one obtains an array that 

s has one or more peaks — four peaks for a rectangle with 
four (possibly incomplete) sides showing. Each peak 
lies at the angle at which a line exists on the original 
edge curve. With rectangle-invariant angles, each peak 
lies at the same rectangle-invariant angle, and this, the 

io rotation angle, is calculated from the peak value in the 
array (step 308) The locations of the line segments of 
the sides of the visible rectangle can be also optionally 
be obtained from the corresponding radii using known 
Hough transform methods. This is an alternative to 

75 steps 140-144 of FIG. 1. 

[0027] As shown in FIG. 1 , after the rotation angle has 
been calculated, the edge curve is rotated by the nega- 
tive of the calculated angle, which, for a rectangular pho- 
tograph, aligns the edges with the horizontal and vertical 

20 axes of the image and of the computational x-y frame of 
reference (step 130). 

[0026] Having calculated the rotation angle, the meth- 
od removes noise points from the edge curve (step 1 40). 
In the absence of noise, the invariant angle (as found, 

2S for example, by the mapping used in step 208) of each 
pair of adjacent points in the edge curve should be ori- 
ented close to the rotation angle - or, after rotation of 
the edge curve, to 0° (invariant). Excluding points that 
differ too much from this angle will remove unwanted 

30 noise and provide a clean edge curve of the border. In 
one implementation, an angle tolerance of 5° was found 
to give good results for rectangular photographs and to 
be a good tradeoff between the loss of boundary points 
and the inclusion of noise points. 

35 [0029] The method now has an edge curve that con- 
tains points along the sides of the photograph with each 
side, in the case of a rectangular photograph, oriented 
along either the vertical or horizontal axis. Having this 
edge curve, the method locates the visible portions of 

40 the sides of the photograph (step 142). This step oper- 
ates on the assumption that the visible segments of the 
sides of the photograph originate from sides of a rectan- 
gle. The implementation of this step focuses on line seg- 
ments rather than comers in order to handle situations 

45 in which corners may not be present within the larger 
image. The larger image may contain all or portions of 
one, two, three, or four sides of the photograph bound- 
ary rectangle. In order to handle all the cases, a two- 
step procedure is used, as shown in FIG. 4. 

so [0030] FIG. 4 shows a method 400 implementing stec 
142 of FIG. 1. First, horizontal and vertical density pro- 
files of the edge curve are calculated (step 402). For 
purposes of illustrating method 400, FIG. 5 shows an 
edge curve 502 tor a rectangular photograph that in the 

55 process of scanning has lost two corners, a full side and 
parts of two opposite sides, and density profiles 504 and 
506 corresponding to edge curve 502. For each density 
profile, the following steps are performed. 
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[0031] If a peak is found in a density profile above a 
threshold level (the yes branch of decision step 404), 
the location of each peak is used as the location of an 
end of the segment (step 408). If no peaks are found or 
only one peak is found (the no branches of decision 
steps 404 and 410), the density profile is collapsed to 
give an indicator profile (step 406). An indicator profile 
has a values that are either 0 or 1 , depending on whether 
the corresponding density profile values are zero or 
nonzero, respectively. Indicator profile 508 (FIG. 5) is 
derived collapsed from density profile 506. Use of an 
indicator profile allows sliding window summation of val- 
ues to provide a window summation curve (for example, 
curve 510) from which edge end locations can be cal- 
culated. 

[0032] The method begins with a window length that 
may be too large and calculates a window summation 
curve (step 420). The height of the resulting peak is 
compared to the window length (decision step 424). In 
general, the larger the window length, the smaller is the 
sensitivity to noise. If the window length is at least as 
long as the edge segment, the peak will be as high as 
the edge segment is long. If the peak height is less than 
the window length, the window length is set to the peak 
height (step 422 from the no branch of step 424). The 
window summation (step 420) is then repeated with the 
new window height. (In an alternative implementation, 
the desired window length is half the segment length, 
so the window length is set to half the peak height in 
step 422.) The end points are finally calculated by locat- 
ing the peak in the window summation curve and finding 
the segment end points at the curve points that have a 
value equal to half the peak value (step 430), that is : the 
mid-points of the ramps that can be seen in curve 510 
of FIG. 5. 

[0033] Having located the edge segments, the quality 
of the results are tested against quality thresholds, 
which may be predetermined or subject to user selec- 
tion, to determine whether to use the embedded image 
that was found or to use the entire, larger image (step 
143, FIG. 1). The thresholds may be set to disable this 
step, so that what was identified as the embedded im- 
age is always used. This step is useful, for example, in 
situations where the image has an area of low contrast 
at the edge (light sky, for example) and a strong interior 
edge (such as a horizon). In one quality test, the larger 
image is partitioned into an inside and an outside of the 
calculated perimeter of the embedded image. If the 
standard deviation of the intensity of the outside is 
above a threshold (for example, if it is above about 10% 
or about 1 5% of the range of possible intensity values), 
or if the standard deviation of the outside exceeds that 
of the inside, the larger image is used as the result of 
the method. This result generally occurs when th em- 
bedded image is coextensive with the larger image and 
has a strong interior edge extending through it. In an- 
other quality test, which may be combined with the first 
or used as an alternative, the number of edge points on 



the calculated edge segments is compared with the 
maximum possible number of edge points given the 
combined lengths of th edge segments. If the popula- 
tion of edge points in the combined edge segments is 
s too sparse -- lor example, less than about 1 0% or about 
15% of the maximum total possible - the entire, larger 
image is treated as the embedded image and used as 
the result. 

[0034] After step 142 or, if it is performed, step 143, 

10 the corners are located from the edge end coordinates 
in the rotated reference frame. Having the rotation angle 
and the positions of the visible edge segments, the co- 
ordinates of the comers of the photograph in the original 
image can readily be calculated (step 144). (Note that 

75 in the case of a photograph that has lost one end of the 
rectangle, for example, the calculated comers will be 
those of a smaller rectangle that covers the visible part 
of the photograph, rather than those of the original phys- 
ical photograph. However, the smaller covering rectan- 

20 gie is sufficient for all purposes of later computation.) 
[0035] Having the comer coordinates, operations can 
be performed on the embedded image. For example, 
the embedded image can be cropped (step 1 46). 
[0036] The invention may be implemented in digital 

25 electronic circuitry, or in computer hardware, firmware, 
software, or in combinations of them. Apparatus of the 
invention may be implemented in a computer program 
product tangibly embodied in a machine-readable stor- 
age device for execution by a programmable processor; 

30 and method steps of the invention may be performed by 
a programmable processor executing a program of in- 
structions to perform functions of the invention by oper- 
ating on input data and generating output. The invention 
may advantageously be implemented in one or more 

35 computer programs that are executable on a program- 
mable system including at least one programmable 
processor coupled to receive data and instructions from, 
and to transmit data and instructions to, a data storage 
system, at least one input device, and at least one output 

40 device. Each computer program may be implemented 
in a high-level procedural or object-oriented program- 
ming language, or in assembly or machine language if 
desired; and in any case, the language may be a com- 
piled or interpreted language. Suitable processors in- 

45 elude, by way of example, both general and special pur- 
pose microprocessors. Generally, a processor will re- 
ceive instructions and data from a read-only memory 
and/or a random access memory. Storage devices suit- 
able for tangibly embodying computer program instruc- 

50 tions and data inciude all forms of non-volatile memory, 
including by way of example semiconductor memory de- 
vices, such as EPROM, EEPROM, and flash memory 
devices; magnetic disks such as internal hard disks and 
removable disks; magn to-optical disks; and CD-ROM 

55 disks. Any of the foregoing may be supplemented by, or 
incorporated in, specially-designed ASICs (application- 
specific integrated circuits). 

[0037] By way of example, as shown in FIG. 6, a scan- 
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ner 650 includes a microprocessor 652 for executing 
program instructions stored on a random access mem- 
ory (RAM) 654 and controlling operation of a scanning 
imager 656, which includes the mechanical and optical 
elements of the scanner, such as a scanner bed and 
imaging optics. The scanner may also include a sepa- 
rate program memory such as a writable read-only 
memory (ROM), for example, a flash ROM. 
[0038] The scanner may be connected to a computer 
600 which includes a processor 602, a random access 
memory (RAM) 604, a program memory, a hard drive 
controller, a video controller for a display device, a dis- 
play device, and an input/output (I/O) controller, coupled 
to each other by one or more data transfer busses. The 
computer 600, the scanner 650, or both may be prepro- 
grammed, in ROM, for example, or it may be pro- 
grammed (and rep rog rammed) by loading a program 
from another source (for example, from a floppy disk, a 
CD-ROM, or another computer). In this way, the com- 
puter and scanner may be configured lo perform steps 
and to be apparatus implementing the techniques de- 
scribed in this specification. 

[0039] For example, FIG. 7 shows a method 700 im- 
plementing these techniques in a flatbed scanner to find 
and scan an image such as a rectangular photograph 
on the bed of the scanner. The method can be imple- 
mented in the scanner with a programmed microproc- 
essor controlling the scanner, as has been described. 
The method can be made available to a user as a mode 
of operation selectable through a user interface of the 
scanner, which may be presented to a user on a scanner 
control panel or through a computer program interface 
created by operation of a program running on a compu- 
ter that is in communication with the scanner. Method 
700 begins by performing a low resolution scan of the 
photograph. Using the raster image resulting from the 
scan and applying the techniques described above, the 
scanner locates a rectangle covering the photograph on 
the bed of the scanner (step 704). If the scanner can 
scan at the rotation angle of the photograph, the scanner 
can scan the covering rectangle at high resolution (step 
708). Otherwise, and generally, the location of a gener- 
ally larger covering rectangle that is aligned with the 
scanner is calculated (step 706). Then the scanner 
scans just that portion of the scanner bed that is covered 
by the covering rectangle (step 708). In this way, a pho- 
tograph may be scanned at a high resolution to produce 
a smaller image file than would be produced by a high 
resolution scan of the entire scanner bed. 
[0040] It should be noted that the operation of the 
methods of finding a covering rectangle described are 
robust and do not require the original embedded image 
to be rectangular or even regular in shape. The methods 
of the invention will find a rotation angle even for an ir- 
regular shape. The angle found will generally be that of 
a best, longest line segment on the perimeter of the 
shape, or of a number of parallel or perpendicular line 
segments. The rotation of the edge curve of the shape 
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may be arbitrary, in the sense that it does not necessarily 
orient the shape to place the image in what would be 
seen as an upright viewing position. Nevertheless, by 
applying method 400 (described in reference to FIG. 4) 
a covering rectangle can be computed within which the 
image, even an irregular image, will be found. 
[0041] Other embodiments are within the scope of the 
following claims. For example, the order of performing 
steps of the invention may be changed by those skilled 
in the art and still achieve desirable results. The shape 
of the photograph may be determined dynamically from 
the edge curve and used to determine dynamically the 
invariant angle mapping. 



Claims 

1 . Apparatus for finding an image embedded in a larg- 
er image, comprising: 

means for finding an edge curve of the embed- 
ded image in the larger image; 
means for calculating a rotation angle of the 
embedded image from the edge curve; and 
means for calculating from the edge curve a lo- 
cation and orientation of one or more line seg- 
ments of a side of a rectangle bounding the em- 
bedded image in the larger image. 



30 2. The apparatus of claim 1 , wherein the embedded 
image is an image of a rectangular photograph, the 
apparatus further comprising: 

means for locating edge segments in the edge 
curve corresponding to sides of the embedded 
image in the larger image. 

3. The apparatus of claim 1 , wherein the embedded 
image has an irregular border shape. 

4. The apparatus of claim 1 , further comprising: 

means for locating corners of a covering rec- 
tangle of the embedded image. 

5. The apparatus of claim 4, further comprising: 

means for cropping the embedded image in a 
frame of reference of the larger image. 

6. The apparatus of claim 4, further comprising: 

means for calculating density profiles on two 
axes of the larger image. 

7. The apparatus of claim 4, further comprising: 
means for detecting edge points. 



35 



40 



45 



50 



55 



SDCCID: <EP 0939381 A2J_> 



11 



EP 0 939 381 A2 



12 



8. The apparatus of claim 7. further comprising: 

means for removing pixel noise from the larger 
image. 

5 

9. A method of processing a digital raster image em- 
bedded in a larger digital raster image, comprising: 

finding an edge curve of the embedded image 
in the larger image; 10 
calculating a rotation angle of the embedded 
image from the edge curve; and 
calculating from the edge curve a location and 
orientation of a line segment of a side of a rec- 
tangle bounding the embedded image in the 15 
larger image. 

10. The method of claim 9, wherein: 

the larger image is a digital raster image gen- 20 
erated by scanning an analog input image. 

11. The method of claim £, further comprising: 

performing a scan at a first resolution of a first 2s 
area to obtain the larger digital raster image; 
and 

using the edge curve to define a second area 
smaller than, and within, the first area, and per- 
forming a scan of the second area at a second 30 
resolution higher than the first resolution to ob- 
tain a raster image including the embedded dig- 
ital raster image at the second resolution. 

12. The method of claim 9, further comprising: 35 

finding the edge curve by: 

calculating the positions of edge points in 
the larger image, and 40 
scanning from the perimeter into the larger 
image from starting points on all sides of 
the larger image and selecting as a point 
in the edge curve the first edge point en- 
counter from each starting point; 45 

calculating a rotation angle by calculating an 
average angle of the edge curve points over a 
windowed set of neighboring edge curve 
points: 50 
rotating the edge curve by the complement of 
the calculated rotation angle; and 
calculating the location of corners of a rectan- 
gle covering the embedded image. 

55 

13. The method of claim 12, wherein: 

the two axes are perpendicular; and 



calculating the positions of edge points com- 
prises applying a Sobel filter to the larger im- 
age. 

14. The method of claim 12, wherein: 

the two axes are perpendicular; and 
calculating the positions of edge points com- 
prises applying a Hough transform to the larger 
image. 

15. The method of claim 1 2, further comprising: 

cropping the embedded image according the 
covering rectangle. 

16. The method of claim 12, further comprising: 

performing a scan at a first resolution of a first 
area to obtain the larger digital raster image; 
and 

using the bounding rectangle to define a sec- 
ond area smaller than, and within, the first area, 
and performing a scan of the second area at a 
second resolution higher than the first resolu- 
tion to obtain a raster image including the em- 
bedded digital raster image at the second res- 
olution. 

17. The method of claim 9, further comprising: 

removing pixel noise from the larger image be- 
fore performing the step of finding an edge 
curve; 

finding the edge curve by 

calculating density profiles of the larger im- 
age on two axes; 

calculating the positions of edge points in 
the larger image; and 
scanning from the perimeter into the larger 
image from starting points on all sides of 
the larger image and selecting as a point 
in the edge curve the first edge point en- 
counter from each starting point; 

calculating a rotation angle by calculating an 
average angle of the edge curve points over a 
windowed set of neighboring edge curve 
points; 

rotating the edge curve by the complement of 
the calculated rotation angle; 
removing noise points from the edge curve; and 
calculating the location of corners of a rectan- 
gle covering the embedded image. 

18. The method of claim 17, further comprising: 
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performing a scan at a first resolution of a first 
area to obtain the larger digital raster image; 
and 

using the bounding rectangle to define a sec- 
ond area smaller than, and within, the first area, 5 
and performing a scan of the second area at a 
second resolution higher than the first resolu- 
tion to obtain a raster image including the em- 
bedded digital raster image at the second res- 
olution. io 

19. The method of claim 18, wherein the scan of the 
second area is limited to the second area. 
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(54) Locating and aligning embedded images 

(57) Apparatus and methods useful to automate the 
task of locating a photograph or other image that is em- 
bedded within a larger image. An edge curve is found 
that approximates the perimeter of the embedded image 
and from the edge curve the rectangle of the embedded 
image, or a rectangle covering the embedded image, is 
calculated by processing density profiles of the edge 
curve taken along two axes. Both location and orienta- 
tion of an embedded image can be calculated. The lo- 
cation of the four comers ol a rectangular embedded im- 
age can be calculated, which enables automatic crop- 
ping and rotation of the embedded image, even if fewer 
than all four comers are visible. A rectangle covering the 
image can be calculated, including a rectangle aligned 
with the axes of a larger embedding image such as is 
created when scanning a small photograph on a flatbed 
scanner. The invention operates well on rectangular im- 
ages, as well as on images having polygonal and other 
border shapes, such as circles and ellipses, including 
images with irregular shapes. 



IMAGE EMBEDDED 
IN IMAGE 



FIG. 1 





^04 


(OPTIONAL) 
REMOVE PIXEL NOISE 




^106 


(OPTIONAL) 
ADJUST IMAGE 






(OPTIONAL) 
CALCULATE DENSITY 
PROFILES ON TWO AXES 






DETECT EC 


•GE POINTS 






FIND EDGE CURVE OF 
EMBEDDED IMAGE 






CALCULATE ROTATION 
ANGLE OF 
EMBEDDED IMAGE 



ROTATE EDGE CURVE BY 
ANGLE TO ALIGN EDGES TO 
HORIZONTAL AND VERTICAL 
AXES 






REMOVE NOISE POINTS 
FROM EDGE CURVE 






LOCATE EDGE SEGMENTS 






TEST QUALITY OF RESULTS 
AND USE EMBEDDING IMAGE 
IF QUALITY BELOW 
THRESHOLD 






LOCATE CORNERS 




S-146 



CROP EMBEDDED IMAGE 



Printed by Jouve, 75001 PARIS (FR) 



MSDOCID: <EP 0939381 A3 J_> 



EP 0 939 381 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 99 30 13B5 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with rdicatjon. where appropriate, 
of relevant passages 



EP 0 660 256 A (CANON KK ; CANON 
INFORMATION SYST INC (US)) 
28 June 1995 (1995-06-28) 

* abstract; figures 1,5,7A,7B * 

* page 2, line 7 - page 3, line 43 * 

EP 0 736 999 A (CANON INFORMATION SYST 
INC) 9 October 1996 (1996-10-09) 

* abstract; figures 3,4,8,13 * 

EP 0 703 696 A (IBM) 

27 March 1996 (1996-03-27) 

* abstract; figure 3 * 



The preseni search report has been drawn up for all claims 



Relevant 
to claim 



1-19 



1-19 



1,9 



CLASSIFICATION OF THE 
APPLICATION (Int. CI .6) 



G06T3/00 



TECHNICAL FIELDS 
SEARCHED (kit. CI. 6) 



606T 

G06K 



MUNICH 



27 October 2000 



Konig, U 



CATEGORY OF CITED DOCUMENTS 

X : particularly relevant If taken alone 

Y : particular!/ retevan . if combined *rt1h another 

document of the same category 
A : technologca! background 
O : non-written disclosure 
P • intermediate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on. or 

after the filing Gate 
D . rlocaimenl cil«t in the application 
L : document cited for other reasons 



& : member of the same patent family, correspond ng 
document 



NSDOCID: <EP 0939381 A3J_> 



2 



EP 0 939 381 A3 



ANNEX TO THE EUROPEAN SEARCH REPORT 
ON EUROPEAN PATENT APPLICATION NO. 



EP 99 30 1355 



This annex lists the patent family members relating to the patent documents cited In the above-mentioned European search report 
The members are as contained In the European Patent Office EDP file on 

The European Patent Office is in no way liable tor these particulars which are merely given for the purpose of information. 

27-10-2000 



Patent document 
cited in search leport 



Publication 
date 



Patent family 
er(s) 



Publication 
dale 



EP 0660256 



28-06-1995 



US 
OP 
US 



5588072 A 
7234918 A 
5854853 A 



EP 0736999 



09-10-1996 



US 6005680 A 
JP 8289094 A 



EP 0703696 



27-03-1996 



US 5517587 A 

CA 2157969 A 

JP 2986383 B 

JP 8123900 A 



24-12-1996 
05-09-1995 
29-12-1998 



21-12-1999 
01-11-1996 



14-05-1996 
24-03-1996 
06-12-1999 
17-05-1996 



o 

Si For more 



about this annex : see Official Journal of the European Patent Office, No 12/82 



ISDOCID: <EP 0939381 A3_l_> 



3 



